{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Задание 3.2 - сверточные нейронные сети (CNNs) в PyTorch\n",
    "\n",
    "Это упражнение мы буде выполнять в Google Colab - https://colab.research.google.com/  \n",
    "Google Colab позволяет запускать код в notebook в облаке Google, где можно воспользоваться бесплатным GPU!  \n",
    "\n",
    "Авторы курса благодарят компанию Google и надеятся, что праздник не закончится.\n",
    "\n",
    "Туториал по настройке Google Colab:  \n",
    "https://medium.com/deep-learning-turkey/google-colab-free-gpu-tutorial-e113627b9f5d  \n",
    "(Keras инсталлировать не нужно, наш notebook сам установит PyTorch)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "FcXBeP1O7cnY"
   },
   "outputs": [],
   "source": [
    "# Intstall PyTorch and download data\n",
    "!pip3 install torch torchvision\n",
    "\n",
    "!wget -c http://ufldl.stanford.edu/housenumbers/train_32x32.mat http://ufldl.stanford.edu/housenumbers/test_32x32.mat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-afwWw-Q85vD"
   },
   "outputs": [],
   "source": [
    "from collections import namedtuple\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import PIL\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import torchvision.datasets as dset\n",
    "from torch.utils.data.sampler import SubsetRandomSampler\n",
    "\n",
    "from torchvision import transforms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NNU-OD9O9ltP"
   },
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda:0\") # Let's make sure GPU is available!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Загружаем данные"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YAvkoRx-9FsP"
   },
   "outputs": [],
   "source": [
    "# First, lets load the dataset\n",
    "data_train = dset.SVHN('./', \n",
    "                       transform=transforms.Compose([\n",
    "                           transforms.ToTensor(),\n",
    "                           transforms.Normalize(mean=[0.43,0.44,0.47],\n",
    "                                               std=[0.20,0.20,0.20])                           \n",
    "                       ])\n",
    "                      )\n",
    "data_test = dset.SVHN('./', split='test', transform=transforms.Compose([\n",
    "                           transforms.ToTensor(),\n",
    "                           transforms.Normalize(mean=[0.43,0.44,0.47],\n",
    "                                               std=[0.20,0.20,0.20])                           \n",
    "                       ]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Разделяем данные на training и validation.\n",
    "\n",
    "На всякий случай для подробностей - https://pytorch.org/tutorials/beginner/data_loading_tutorial.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YRnr8CPg7Hli"
   },
   "outputs": [],
   "source": [
    "batch_size = 64\n",
    "\n",
    "data_size = data_train.data.shape[0]\n",
    "validation_split = .2\n",
    "split = int(np.floor(validation_split * data_size))\n",
    "indices = list(range(data_size))\n",
    "np.random.shuffle(indices)\n",
    "\n",
    "train_indices, val_indices = indices[split:], indices[:split]\n",
    "\n",
    "train_sampler = SubsetRandomSampler(train_indices)\n",
    "val_sampler = SubsetRandomSampler(val_indices)\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(data_train, batch_size=batch_size, \n",
    "                                           sampler=train_sampler)\n",
    "val_loader = torch.utils.data.DataLoader(data_train, batch_size=batch_size,\n",
    "                                         sampler=val_sampler)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LyYvt-T67PBG"
   },
   "outputs": [],
   "source": [
    "# We'll use a special helper module to shape it into a flat tensor\n",
    "class Flattener(nn.Module):\n",
    "    def forward(self, x):\n",
    "        batch_size, *_ = x.shape\n",
    "        return x.view(batch_size, -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Создадим простейшую сеть с новыми слоями:  \n",
    "Convolutional - `nn.Conv2d`  \n",
    "MaxPool - `nn.MaxPool2d`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "w9SFVGZP7SQd"
   },
   "outputs": [],
   "source": [
    "nn_model = nn.Sequential(\n",
    "            nn.Conv2d(3, 64, 3, padding=1),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.MaxPool2d(4),\n",
    "            nn.Conv2d(64, 64, 3, padding=1),\n",
    "            nn.ReLU(inplace=True),\n",
    "            nn.MaxPool2d(4),    \n",
    "            Flattener(),\n",
    "            nn.Linear(64*2*2, 10),\n",
    "          )\n",
    "\n",
    "nn_model.type(torch.cuda.FloatTensor)\n",
    "nn_model.to(device)\n",
    "\n",
    "loss = nn.CrossEntropyLoss().type(torch.cuda.FloatTensor)\n",
    "optimizer = optim.SGD(nn_model.parameters(), lr=1e-1, weight_decay=1e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Восстановите функцию `compute_accuracy` из прошлого задания.  \n",
    "Единственное отличие в новом - она должна передать данные на GPU прежде чем прогонять через модель. Сделайте это так же, как это делает функция `train_model`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2ek3KVQK7hJ6"
   },
   "outputs": [],
   "source": [
    "def train_model(model, train_loader, val_loader, loss, optimizer, num_epochs):    \n",
    "    loss_history = []\n",
    "    train_history = []\n",
    "    val_history = []\n",
    "    for epoch in range(num_epochs):\n",
    "        model.train() # Enter train mode\n",
    "        \n",
    "        loss_accum = 0\n",
    "        correct_samples = 0\n",
    "        total_samples = 0\n",
    "        for i_step, (x, y) in enumerate(train_loader):\n",
    "          \n",
    "            x_gpu = x.to(device)\n",
    "            y_gpu = y.to(device)\n",
    "            prediction = model(x_gpu)    \n",
    "            loss_value = loss(prediction, y_gpu)\n",
    "            optimizer.zero_grad()\n",
    "            loss_value.backward()\n",
    "            optimizer.step()\n",
    "            \n",
    "            _, indices = torch.max(prediction, 1)\n",
    "            correct_samples += torch.sum(indices == y_gpu)\n",
    "            total_samples += y.shape[0]\n",
    "            \n",
    "            loss_accum += loss_value\n",
    "\n",
    "        ave_loss = loss_accum / i_step\n",
    "        train_accuracy = float(correct_samples) / total_samples\n",
    "        val_accuracy = compute_accuracy(model, val_loader)\n",
    "        \n",
    "        loss_history.append(float(ave_loss))\n",
    "        train_history.append(train_accuracy)\n",
    "        val_history.append(val_accuracy)\n",
    "        \n",
    "        print(\"Average loss: %f, Train accuracy: %f, Val accuracy: %f\" % (ave_loss, train_accuracy, val_accuracy))\n",
    "        \n",
    "    return loss_history, train_history, val_history\n",
    "        \n",
    "def compute_accuracy(model, loader):\n",
    "    \"\"\"\n",
    "    Computes accuracy on the dataset wrapped in a loader\n",
    "    \n",
    "    Returns: accuracy as a float value between 0 and 1\n",
    "    \"\"\"\n",
    "    model.eval() # Evaluation mode\n",
    "    # TODO: Copy implementation from previous assignment\n",
    "    # Don't forget to move the data to device before running it through the model!\n",
    "    \n",
    "    raise Exception(\"Not implemented\")\n",
    "\n",
    "loss_history, train_history, val_history = train_model(nn_model, train_loader, val_loader, loss, optimizer, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6a-3a1ZFGEw_"
   },
   "source": [
    "# Аугментация данных (Data augmentation)\n",
    "\n",
    "В работе с изображениями одним из особенно важных методов является аугментация данных - то есть, генерация дополнительных данных для тренировки на основе изначальных.   \n",
    "Таким образом, мы получаем возможность \"увеличить\" набор данных для тренировки, что ведет к лучшей работе сети.\n",
    "Важно, чтобы аугментированные данные были похожи на те, которые могут встретиться в реальной жизни, иначе польза от аугментаций уменьшается и может ухудшить работу сети.\n",
    "\n",
    "С PyTorch идут несколько таких алгоритмов, называемых `transforms`. Более подробно про них можно прочитать тут -\n",
    "https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#transforms\n",
    "\n",
    "Ниже мы используем следующие алгоритмы генерации:\n",
    "- ColorJitter - случайное изменение цвета\n",
    "- RandomHorizontalFlip - горизонтальное отражение с вероятностью 50%\n",
    "- RandomVerticalFlip - вертикальное отражение с вероятностью 50%\n",
    "- RandomRotation - случайный поворот"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jCWMUWmr7t5g"
   },
   "outputs": [],
   "source": [
    "tfs = transforms.Compose([\n",
    "    transforms.ColorJitter(hue=.50, saturation=.50),\n",
    "    transforms.RandomHorizontalFlip(),\n",
    "    transforms.RandomVerticalFlip(),\n",
    "    transforms.RandomRotation(50, resample=PIL.Image.BILINEAR),\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize(mean=[0.43,0.44,0.47],\n",
    "                       std=[0.20,0.20,0.20])                           \n",
    "])\n",
    "\n",
    "# Create augmented train dataset\n",
    "data_aug_train = dset.SVHN('./', \n",
    "                       transform=tfs\n",
    "                      )\n",
    "\n",
    "train_aug_loader = torch.utils.data.DataLoader(data_aug_train, batch_size=batch_size, \n",
    "                                           sampler=train_sampler)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Визуализируем результаты агментации (вообще, смотреть на сгенерированные данные всегда очень полезно)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YlJJEro1KZ45"
   },
   "outputs": [],
   "source": [
    "# TODO: Visualize some augmented images!\n",
    "# hint: you can create new datasets and loaders to accomplish this\n",
    "\n",
    "# Based on the visualizations, should we keep all the augmentations?\n",
    "\n",
    "tfs = transforms.Compose([\n",
    "    transforms.ColorJitter(hue=.20, saturation=.20),\n",
    "    transforms.RandomHorizontalFlip(),\n",
    "    transforms.RandomVerticalFlip(),\n",
    "    transforms.RandomRotation(10, resample=PIL.Image.BILINEAR),\n",
    "])\n",
    "\n",
    "data_aug_vis = dset.SVHN('./', \n",
    "                       transform=tfs\n",
    "                      )\n",
    "\n",
    "plt.figure(figsize=(30, 3))\n",
    "\n",
    "for i, (x, y) in enumerate(data_aug_vis):\n",
    "    if i == 10:\n",
    "        break\n",
    "    plt.subplot(1, 10, i+1)\n",
    "    plt.grid(False)\n",
    "    plt.imshow(x)\n",
    "    plt.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "o2LrmsYHoguB"
   },
   "source": [
    "Все ли агментации одинаково полезны на этом наборе данных? Могут ли быть среди них те, которые собьют модель с толку?\n",
    "\n",
    "Выберите из них только корректные"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "evro9ksXGs9u"
   },
   "outputs": [],
   "source": [
    "# TODO: \n",
    "tfs = transforms.Compose([\n",
    "    # TODO: Add good augmentations\n",
    "    transforms.ToTensor(),\n",
    "    transforms.Normalize(mean=[0.43,0.44,0.47],\n",
    "                       std=[0.20,0.20,0.20])                           \n",
    "])\n",
    "\n",
    "# TODO create new instances of loaders with the augmentations you chose\n",
    "train_aug_loader = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PeO6Zw0DHqPR"
   },
   "outputs": [],
   "source": [
    "# Finally, let's train with augmentations!\n",
    "\n",
    "# Note we shouldn't use augmentations on validation\n",
    "\n",
    "loss_history, train_history, val_history = train_model(nn_model, train_aug_loader, val_loader, loss, optimizer, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "r0bcioK6JBDK"
   },
   "source": [
    "# LeNet\n",
    "Попробуем имплементировать классическую архитектуру сверточной нейронной сети, предложенную Яном ЛеКуном в 1998 году. В свое время она достигла впечатляющих результатов на MNIST, посмотрим как она справится с SVHN?\n",
    "Она описана в статье [\"Gradient Based Learning Applied to Document Recognition\"](http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf), попробуйте прочитать ключевые части и имплементировать предложенную архитетуру на PyTorch.\n",
    "\n",
    "Реализовывать слои и функцию ошибки LeNet, которых нет в PyTorch, **не нужно** - просто возьмите их размеры и переведите в уже известные нам Convolutional, Pooling и Fully Connected layers.\n",
    "\n",
    "Если в статье не очень понятно, можно просто погуглить LeNet и разобраться в деталях :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ieEzZUglJAUB"
   },
   "outputs": [],
   "source": [
    "# TODO: Implement LeNet-like architecture for SVHN task\n",
    "lenet_model = nn.Sequential(\n",
    "          )\n",
    "\n",
    "lenet_model.type(torch.cuda.FloatTensor)\n",
    "lenet_model.to(device)\n",
    "\n",
    "loss = nn.CrossEntropyLoss().type(torch.cuda.FloatTensor)\n",
    "optimizer = optim.SGD(lenet_model.parameters(), lr=1e-1, weight_decay=1e-4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WMmaPfdeKk9H"
   },
   "outputs": [],
   "source": [
    "# Let's train it!\n",
    "loss_history, train_history, val_history = train_model(lenet_model, train_aug_loader, val_loader, loss, optimizer, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "u_O9qiYySvuj"
   },
   "source": [
    "# Подбор гиперпараметров"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "i6mhfdQ9K-N3"
   },
   "outputs": [],
   "source": [
    "# The key hyperparameters we're going to tune are learning speed, annealing rate and regularization\n",
    "# We also encourage you to try different optimizers as well\n",
    "\n",
    "Hyperparams = namedtuple(\"Hyperparams\", ['learning_rate', 'anneal_epochs', 'reg'])\n",
    "RunResult = namedtuple(\"RunResult\", ['model', 'train_history', 'val_history', 'final_val_accuracy'])\n",
    "\n",
    "learning_rates = [1e0, 1e-1, 1e-2, 1e-3, 1e-4]\n",
    "anneal_coeff = 0.2\n",
    "anneal_epochs = [1, 5, 10, 15, 20, 50]\n",
    "reg = [1e-3, 1e-4, 1e-5, 1e-7]\n",
    "\n",
    "batch_size = 64\n",
    "epoch_num = 10\n",
    "\n",
    "# Record all the runs here\n",
    "# Key should be Hyperparams and values should be RunResult\n",
    "run_record = {} \n",
    "\n",
    "# Use grid search or random search and record all runs in run_record dictionnary \n",
    "# Important: perform search in logarithmic space!\n",
    "\n",
    "# TODO: Your code here!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "Y6xExdw8JB1l",
    "outputId": "a9ad86f8-3e29-45cc-d33f-e6170018a4ed"
   },
   "outputs": [],
   "source": [
    "best_val_accuracy = None\n",
    "best_hyperparams = None\n",
    "best_run = None\n",
    "\n",
    "for hyperparams, run_result in run_record.items():\n",
    "    if best_val_accuracy is None or best_val_accuracy < run_result.final_val_accuracy:\n",
    "        best_val_accuracy = run_result.final_val_accuracy\n",
    "        best_hyperparams = hyperparams\n",
    "        best_run = run_result\n",
    "        \n",
    "print(\"Best validation accuracy: %4.2f, best hyperparams: %s\" % (best_val_accuracy, best_hyperparams))\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LOmsR0uVgtgf"
   },
   "source": [
    "# Свободное упражнение - догоним и перегоним LeNet!\n",
    "\n",
    "Попробуйте найти архитектуру и настройки тренировки, чтобы выступить лучше наших бейзлайнов.\n",
    "\n",
    "Что можно и нужно попробовать:\n",
    "- BatchNormalization (для convolution layers он в PyTorch называется [batchnorm2d](https://pytorch.org/docs/stable/nn.html#batchnorm2d))\n",
    "- Изменить количество слоев и их толщину\n",
    "- Изменять количество эпох тренировки\n",
    "- Попробовать и другие агментации"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tSVhD747icoc"
   },
   "outputs": [],
   "source": [
    "best_model = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ubeKgBcnhx7N"
   },
   "source": [
    "# Финальный аккорд - проверим лучшую модель на test set\n",
    "\n",
    "В качестве разнообразия - напишите код для прогона модели на test set вы.\n",
    "\n",
    "В результате вы должны натренировать модель, которая покажет более **90%** точности на test set.  \n",
    "Как водится, лучший результат в группе получит дополнительные баллы!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EIqM1kdeh-hd"
   },
   "outputs": [],
   "source": [
    "# TODO Write the code to compute accuracy on test set\n",
    "final_test_accuracy = 0.0\n",
    "print(\"Final test accuracy - \", final_test_accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BfH6qip6kVX_"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "PyTorch_CNN.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
